Bug 1949827: Add KUBELET_NODEIP_HINT to nodeip-configuration #2888
Conversation
openshift-ci
bot
added
bugzilla/severity-urgent
|
@cybertron: This pull request references Bugzilla bug 1949827, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Bugzilla ([email protected]), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cc @danwinship This is essentially what you suggested as a workaround on the bz. I went with a more generic env var name in case some of these deployments did not want the nodes listening on the same subnet as the api server (although...is that even valid?). I also used the built-in override mechanism in systemd to specify the value which kept the changes to the service file minimal. I semi-successfully tested this in an IPI deployment by hacking up the nodeip-configuration for IPI to do the same thing. It seemed to work, although I couldn't complete deployment that way because my bootstrap node wasn't connected to the additional network I specified so there was no connectivity with the nodes. They did select the correct address to bind to though. |
|
/retest-required |
|
/retest-required |
I think so, though it would be weird...
I used rather than: |
|
/lgtm |
Hmm, yeah. I guess that must have been a holdover from when I tried to implement this only in baremetal-runtimecfg (which didn't work because podman doesn't pass env vars through by default, so service changes were needed anyway). Let me try switching this to EnvironmentFile since there's no reason to do it this way now. |
|
@cybertron: This pull request references Bugzilla bug 1949827, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Bugzilla ([email protected]), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Okay, I switched to EnvironmentFile. I had to add a default one that sets the variable to empty or the service fails when you don't create the file. Otherwise this is pretty much the same as before. |
|
Okay, tested this version locally and it works. I think it should be good to go. |
|
/lgtm |
|
@cybertron can you make sure to get the ocp docs updated to include this (if it doesn't already)? |
When we added the nodeip-configuration service for None platform deployments, we broke some existing users who were relying on the (largely undefined) previous behavior Kubelet used to select its node ip. While it is possible to work around this by overriding the node ip selection logic, that's very cumbersome and not an acceptable user experience. This change adds a KUBELET_NODEIP_HINT env variable that can be used to override the default behavior of runtimecfg when selecting a node ip. When the variable is unset, the old behavior of selecting an address on the interface of the default route will take effect. When the variable is set, its value will be passed to runtimecfg like a VIP for the IPI platforms. This will cause runtimecfg to prefer an address in the same subnet as the one provided in KUBELET_NODEIP_HINT. If no such address is found, it will fall back to the default route logic as before. KUBELET_NODEIP_HINT can be set using a systemd environment file. The file must be named /etc/default/nodeip-configuration with contents such as (replacing the IP as appropriate): KUBELET_NODEIP_HINT=192.0.2.1 This file should be created using a machine-config manifest that is passed to the installer so it will take effect on initial deployment. The node ip cannot be changed after the node registers initially so this cannot be done as a day 2 operation. Note that the IP specified in the hint does not necessarily need to exist in the environment, it just needs to be in the correct subnet. No traffic will be sent to this address. Co-authored-by: Dan Winship <[email protected]>
cybertron
force-pushed
the
nodeip-hint
branch
from
3afc4be to
a0c9a3c
Compare
|
Squashed the changes and I'll talk to the docs folks about where to put this. |
|
/bugzilla refresh The requirements for Bugzilla bugs have changed (BZs linked to PRs on master branch need to target OCP 4.11), recalculating validity. |
openshift-ci
bot
removed
the
bugzilla/valid-bug
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
13 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
@cybertron: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
@cybertron: All pull requests linked via external trackers have merged: Bugzilla bug 1949827 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-4.10 |
|
@cybertron: new pull request created: #3058 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
As OpenShift is deployed in increasingly complex networking environments, we have gotten many requests for more control over which interface is used for the primary node IP. We provided a basic mechanism for this with [KUBELET_NODEIP_HINT](openshift/machine-config-operator#2888) but as users have started to exercise that some significant limitations have come to light.
When we added the nodeip-configuration service for None platform
deployments, we broke some existing users who were relying on the
(largely undefined) previous behavior Kubelet used to select its
node ip. While it is possible to work around this by overriding the
node ip selection logic, that's very cumbersome and not an acceptable
user experience.
This change adds a KUBELET_NODEIP_HINT env variable that can be used
to override the default behavior of runtimecfg when selecting a node
ip. When the variable is unset, the old behavior of selecting an
address on the interface of the default route will take effect. When
the variable is set, its value will be passed to runtimecfg like a
VIP for the IPI platforms. This will cause runtimecfg to prefer an
address in the same subnet as the one provided in
KUBELET_NODEIP_HINT. If no such address is found, it will fall back
to the default route logic as before.
KUBELET_NODEIP_HINT can be set using a systemd environment file.
The file must be named /etc/default/nodeip-configuration
with contents such as (replacing the IP as appropriate):
KUBELET_NODEIP_HINT=192.0.2.1
This file should be created using a machine-config manifest that is
passed to the installer so it will take effect on initial deployment.
The node ip cannot be changed after the node registers initially so
this cannot be done as a day 2 operation.
Note that the IP specified in the hint does not necessarily need to
exist in the environment, it just needs to be in the correct subnet.
No traffic will be sent to this address.
- What I did
- How to verify it
- Description for the changelog
Added a mechanism for deployers to override the default node ip selection logic in None platform deployments.